Image processing method, system and computer program to improve an image sensed by an image sensing apparatus and processed according to a conversion process

ABSTRACT

The present invention relates to an image processing method, apparatus and computer program for improving an image sensed by an image sensing apparatus and processed according to a first conversion process. The present invention involves determining whether or not the first conversion process includes a nonlinear conversion, processing the image according to a second conversion process inverse to the first conversion process if the first conversion process includes the nonlinear conversion, and processing the image processed according to the second conversion process, according to a function for improving the image.

BACKGROUND OF THE INVENTION

The present invention relates to an image processing apparatus, imageprocessing method, image sensing apparatus, control method for the imagesensing apparatus, and a memory and, more particularly, to an imageprocessing apparatus and method which improve a defective image(degraded image), image sensing apparatus suitably combined with theimage processing apparatus, control method for the image sensingapparatus, and memory for controlling the execution of the methods. Inthis specification, an image called a degraded image indicates apoor-quality image that is out of focus or blurred due to inappropriatemovement of a camera or inappropriate image sensing conditions such asan exposure instead of indicating an image having undergone a changefrom a good image to a degraded image due to a change in quality overtime or the like.

As methods of improving a degraded image, e.g., an out-of-focus image orblurred image, into an image with little degradation (ideal image),methods using a Wiener filter, general inverted filter, projectionfilter, and the like are available. To use these methods, a degradationfunction must be determined first. An ideal method of determining such adegradation function is a method of analytically obtaining a degradationfunction from physical factors such as image sensing conditions orestimating a degradation function on the basis of an output from ameasuring device (e.g., an acceleration sensor) mounted in an imagesensing apparatus.

A degradation function will be described below. The relationship betweenan ideal image f(x, y), a degraded image g(x, y), a degradation functionh(x, y, x′, y′), and random noise on output image ν(x, y) is expressedas

∫∫h(x,y,x′,y′)f(x′,y′)dx′dy′+ν(x,y)  (1)

If an image having a degraded point is not located at the point, exceptfor translation, a point spread function (PSF) is expressed by h(x−x′,y−y′), and mathematical expression (1) is rewritten into

∫∫h(x−x′,y−y′)f(x′,y′)dx′dy′+ν(x,y)  (2)

If there is no noise, a Fourier transform of the two sides ofmathematical expression (2) is performed, and a convolution theorem isapplied to the resultant expression, equation (3) is obtained:

G(u,v)=H(u,v)F(u,v)  (3)

where G(u, V) H(u, v), and F(u, V) are the Fourier transforms of g(x,y), f(x, y), and h(x, y).

H(u, v) is the transfer function of a system for transforming the idealimage f(x, y) into the degraded image g(x, y).

A degradation model in degradation (blur) due to a relative movementbetween a camera and a scene (object) will be described below as anexample. Assume that an image on the image sensing element of the cameraremains unchanged over time except this relative movement. If therelative movement is approximately equal to the movement of the imagesensing element in the same plane, the total exposure light amount atone point on the image sensing element can be obtained by integrating aninstantaneous exposure light amount with respect to an exposure time.Assume that the time required to open/close the shutter can beneglected. Letting α(t) and β(t) be the x and y components of thedisplacement, equation (4) can be established:

g(x,y)=∫_(−T/2) ^(T/2) f(x−α(t),y−β(t))dt  (4)

where T is the exposure time, and the integration range is set fromt=−T/2 to t=T/2 for the sake of convenience.

A Fourier transform of the two sides of equation (4) yields equation(5): $\begin{matrix}{{G\left( {u,v} \right)} = {{\int{{x}{\int{{y}\quad {\exp \left\lbrack {{- {j2}}\quad {\pi \left( {{ux} - {vy}} \right)}} \right\rbrack}{\int_{{- T}/2}^{T/2}\quad {{{tf}\left( {{x - {\alpha (t)}},{y - {\beta (t)}}} \right)}}}}}}} = {\int_{{- T}/2}^{T/2}\quad {{t}{\int{{x}{\int{{{{yf}\left( {{x - {\alpha (t)}},{y - {\beta (t)}}} \right)}}{\exp \left\lbrack {{- {j2}}\quad {\pi \left( {{ux} - {vy}} \right)}} \right\rbrack}}}}}}}}} & (5)\end{matrix}$

If x−α(t)=ξ and y−β(t)=η, equation (5) is rewritten into equation (6):$\begin{matrix}{{G\left( {u,v} \right)} = {{\int{{x}{\int{\int{{\xi}{\eta}\quad {f\left( {\xi,\eta} \right)} \times {\exp \left\lbrack {{- {j2}}\quad {\pi \left( {{u\quad \xi} - {v\quad \eta}} \right)}} \right\rbrack}{\exp \left\lbrack {{- {j2}}\quad {\pi \left( {{{\alpha (t)}u} + {{\beta (t)}v}} \right)}} \right\rbrack}}}}}} = {{{F\left( {u,v} \right)}{\int_{{- T}/2}^{T/2}{{\exp \left\lbrack {{- {j2}}\quad {\pi \left( {{u\quad {\alpha (t)}} + {v\quad {\beta (t)}}} \right)}} \right\rbrack}{t}}}} = {{F\left( {u,v} \right)}{H\left( {u,v} \right)}}}}} & (6)\end{matrix}$

According to equation (6), the degradation is modeled by equation (3) ormathematical expression (2) which is equivalent to equation (3). Thetransfer function H(u, v) for this degradation is given by

H(u, v)=∫_(−T/2) ^(T/2) exp[−j 2π( uα(t)+vβ(t))]dt  (7)

In this case, if camera shake occurs in a direction at an angle θ withrespect to the x-axis at a predetermined speed V for a time T, a pointresponse function is given as $\begin{matrix}{{H\left( {u,v} \right)} = \frac{\sin \quad {\pi\omega}\quad T}{\pi\omega}} & (8)\end{matrix}$

where ω is given by equation (9)

ω—(u−u _(o))V cos θ+(v−v _(o))V sin θ  (9)

where u_(o) and v_(o) are the center coordinates of the image. When ω isminimum, H(u, v)=T is approximately established.

Likewise, a degradation model of degradation due to a blur can beexpressed by a function. Assume that a phenomenon of blurring is basedon a normal (Guassian) distribution rule. In this case, letting r be thedistance from a central pixel and σ² is an arbitrary parameter in thenormal distribution rule, a degradation function h(r) is given by$\begin{matrix}{{h(r)} = {\frac{1}{\sigma \sqrt{2\pi}}{\exp \left( {- \frac{r^{2}}{\sigma^{2}}} \right)}}} & (10)\end{matrix}$

Processing for improving a degraded image using an inverted filter willbe described next. Assume that the degraded image g(x, y) and the idealimage f(x, y) are based on the model expressed by mathematicalexpression (2). If there is no noise, the Fourier transforms of g(x, y),f(x, y), PSF, and h(x, y) satisfy equation (3). In this case, equation(3) is modified into

F(u,v)=G(u,v)/H(u,v)  (11)

According to equation (11), if H(u, v) is known, the ideal image f (x,y) can be improved by multiplying the Fourier transform G(u, v) of thedegraded image by 1/H(u, v) and performing an inverse Fourier transformof the product. In other words, the transfer function of the filter is1/H(u, v).

In practice, the application of equation (3) poses various problems. Forexample, in consideration of noise, mathematical expression (2) can bewritten into

G(u,v)=H(u,v)F(u,v)+N(u,v)  (12)

where N(u, v) is the Fourier transform of ν(x, y).

According to equation (12), when the filter (1/H) (u, v)) is applied tothe Fourier transform of the degraded image, equation (13) isestablished: $\begin{matrix}{\frac{G\left( {u,v} \right)}{H\left( {u,v} \right)} = {{F\left( {u,v} \right)} + \frac{N\left( {u,v} \right)}{H\left( {u,v} \right)}}} & (13)\end{matrix}$

Consider a system in which the degraded image recorded by the digitalcamera is loaded into an information processing apparatus by an imagereceiving unit controlled by a TWAIN driver or the like, and thedegraded image is improved to generate an ideal image. In this case, atechnique of determining a degradation function obtained by modeling theprocess of generating a degraded image, and improving the degraded imageby using an image improving algorithm generally called deconvolutionusing a Wiener filter or the like is considered as the most effectiveimproving technique.

In such a conventional technique, however, since no consideration isgiven to degradation parameters (e.g., a shake direction, shake speed,and the like if degradation is caused by camera shake) required todetermine a degradation function and image sensing conditions (anexposure time, exposure light amount, and the like), a sufficientimproving effect for a degraded image cannot be obtained.

Many studies have been made on techniques of obtaining a degradationparameter by estimating a degradation state from the image featureamount (e.g., an auto-correlation function) of a degraded image. Amethod of estimating a line spread function, edge spread function, orthe like instead of the above point spread function is also available.For example, the following method of determining a degradation functionis known. In this method, if an acute edge portion is present in anoriginal image, the edge portion is differentiated to obtain a linespread function, and a degradation function is determined by using animage reconstruction method.

If, however, an image is to be improved by using an image improvingalgorithm using a Wiener filter or the like, only several percent of anerror included in a parameter will cause large noise to be superimposedon an improved image. For example, in the technique of obtaining adegradation parameter by estimating a degradation state from the imagefeature amount (e.g., an auto-correction function) of a degraded image,since the parameter often includes a large error, the improving effectis low. Even if a measuring device (acceleration sensor or the like)mounted in an image sensing apparatus such as a camera is to be used,since it is technically very difficult to suppress an error withinseveral percent, the problem of a parameter error cannot be neglected.

As described above, in the prior art, no consideration is given todegradation parameters, image sensing conditions, and the like which arerequired to determine a degradation function, and a degradationparameter cannot be accurately estimated. For this reason, even if adegraded image is improved, the improving effect is not sufficient.

In the technique of using an edge spread function to estimate adegradation parameter, it is difficult to extract an edge portion,because the original image is degraded. The present applicant hasproposed a technique (Japanese Patent Laid-Open No. 7-121703) ofdesignating a small area, of a degraded image, which includes an edgeportion, improving the image within the small area while changing aparameter of a degradation function, obtaining the degree of imageimprovement corresponding to the parameter, and improving the entireimage by using a parameter with which the degree of image improvementbecomes the highest. This proposal, however, can be applied to onlyestimation of an edge spread function but cannot be applied toestimation of a point spread function.

An edge portion of an original image corresponds to a high-frequencycomponent in terms of frequency. When the image is to be improved byusing various filters, the edge portion is an area on which the largestnoise is superimposed. Even if, therefore, a parameter with which thedegree of image improvement becomes the highest is obtained, thepossibility of a low precision due to noise is high. In addition, it iscumbersome to designate a small area, of an original image, whichincludes an edge portion every time improving operation is performed.

In general, in a system including a digital camera, an image is loadedinto an information processing apparatus by an image receiving unitcontrolled by a TWAIN driver or the like, and the resultant image isoutput to an image output apparatus (e.g., a printer or display). Inthis case, the image processing controlled by the TWAIN driver or thelike generally includes gamma correction, color conversion, and thelike. However, gamma correction, color conversion, and the like belongto a nonlinear system and contradict an LTI (Linear Time-Invariant)system serving as a precondition for the above image improvingalgorithm. That is, when a degraded image having undergone gammacorrection or color conversion is improved, very large noise issuperimposed on the improved image.

Consider a system for converting a degraded image recorded on a silverhalide film into an electronic image by using an image input device suchas a film scanner, and improving the degraded image by arithmeticoperation. In general, such a system also uses a technique of forming adegradation function by modeling the process of generating a degradedimage and improving the image by using an image improving algorithmgenerally called deconvolution using a Wiener filter or the like. Thistechnique is regarded as the most effective improving technique.

In practice, however, the electronic image information has been affectedby the aberrations of the lenses of the image sensing apparatus, and theluminance value has been converted by the characteristics (ISO, film γ,and the like) of the film itself, the gamma characteristics of thephotoelectric conversion system of the image input device such as a filmscanner, and the like.

The above lens aberrations, film characteristics, and scannercharacteristics (so-called scanner γ and the like) also generally belongto a nonlinear system and contradict an LTI (Linear Time-Invariant)system as a precondition for the image improving algorithm describedabove. That is, when the degraded image obtained in the above manner isimproved by deconvolution, very large noise is superimposed on theimproved image.

Even if an extremely ideal improved image is generated, noisesuperimposed on the improved image cannot be completely eliminated. Inaddition, since the improved image has no correlation with the colorgradation and luminance distribution of the original image, the improvedimage cannot become an image that is worth looking at without anyprocessing.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the prior art,and has as its object to provide an image processing method andapparatus for improving an image with a high precision while keeping thecomputation load low in improving an image using various degradationfunctions including degradation parameters even if no estimated value isavailable for each degradation parameter or an error is included in thedegradation parameter obtained by analytical estimation or from anoutput from a measuring device mounted in the image sensing apparatus.

The present invention has been made in consideration of the abovesituation, and has as its object to obtain a high-quality improved imagefrom a degraded image.

It is another object of the present invention to provide an imageimproving system and method which improve a degraded image by usingdegradation information required to increase the improving effect whilesuppressing noise to a minimum, and perform image processing for theimage to finally obtain an image that is worth looking at.

It is still another object of the present invention to provide an imageimproving method and system for improving an image with a high precisionwhile keeping the computation load low in improving an image usingvarious degradation functions including degradation parameters even ifno estimated value is available for each degradation parameter or anerror is included in the degradation parameter obtained by analyticalestimation or from an output from a measuring device mounted in theimage sensing apparatus.

In one aspect, the present invention relates to an image processingmethod, apparatus and computer program for improving an image sensed byan image sensing apparatus and processed according to a first conversionprocess. The method, apparatus and computer program involve determiningwhether or not the first conversion process includes a nonlinearconversion, processing the image according to a second conversionprocess inverse to the first conversion process if the first conversionprocess includes the nonlinear conversion, and processing the imageprocessed according to the second conversion process, according to afunction for improving the image.

According to an aspect of the present invention, there is provided animage processing method of improving an image sensed by an image sensingapparatus, comprising:

the input step of inputting image data representing the image throughinput means;

the inversion step of performing, for the image, processing inverse toconversion processing performed for the input image data; and

the improving step of performing improving processing for the imagehaving undergone the inversion processing on the basis of a degradationfunction in image sensing operation.

Preferably, the inversion step further comprises the acquisition step ofacquiring information on the conversion processing performed for theinput image data.

The acquisition step preferably comprises looking up a table in which atype of the input means is made to correspond to a conversion processingmethod used by the input means, and acquiring information on theconversion processing performed for the image data on the basis of thetype of the input means.

The improving step preferably comprises obtaining a degradation functionon the basis of characteristic information on the image sensingapparatus, and performing improving processing for the image data on thebasis of the degradation function.

The improving step preferably comprises looking up a table in which atype of an image sensing apparatus for sensing an image is made tocorrespond to characteristics of the apparatus, and acquiring thecharacteristic information on the basis of the type of image sensingapparatus.

The conversion processing is preferably nonlinear conversion processing.

The nonlinear conversion processing preferably includes gray scaleconversion.

Preferably, the image data has undergone one of conversion processingincluding a nonlinear conversion and conversion processing including nononlinear conversion in accordance with a conversion mode, the inputstep comprises inputting the conversion mode together with image data,and the inversion step comprises referring to information of theconversion mode and performing inversion processing for the image dataif the image data has undergone a conversion including a nonlinearconversion.

Preferably, the input step comprises inputting, together with imagedata, a degradation parameter which is recorded by the image sensingmeans together with an image and indicates a physical quantity of anevent as a cause of degradation, and the improving step comprisesspecifying a degradation function on the basis of the degradationparameter, and performing improving processing for the image inaccordance with the degradation function.

Preferably, the input step comprises inputting, together with imagedata, an image sensing condition, which is recorded by the image sensingmeans together with an image, and the improving step comprisesspecifying a degradation function on the basis of the image sensingcondition and performing improving processing for the image data inaccordance with the degradation function.

Preferably, the method further comprises the post-processing step ofperforming, for the image data improved in the improving step, inversionprocessing to the inversion processing performed for the image data inthe inversion step.

Preferably, the post-processing step further comprises determining adegree of degradation on the basis of the degradation parameter, andperforming noise removal processing for the image data if it isdetermined that the degree of degradation is high.

Preferably, the image sensing apparatus electronically records lightfrom an object to be image-sensed as image data by photoelectricconversion, and the input step comprises inputting the electronicallyrecorded image data.

Preferably, the image sensing apparatus optically records an image on asilver halide film, and the input step comprises inputting the imagedata by using input means for photoelectrically converting the image onthe silver halide film.

Preferably, the image sensing apparatus records characteristicinformation of the image sensing apparatus and/or an image sensingcondition and/or a degradation parameter on a magnetic mediumcorresponding to a silver halide film, and the input step comprisesinputting the pieces of information together with image data.

According to the second aspect of the present invention, there isprovided an image processing method of improving an image havingsuffered degradation in image sensing operation by using a degradationfunction for quantifying the degradation, comprising:

the first improving step of improving the image by using the degradationfunction while changing a degradation parameter included in thedegradation function;

the improvement degree calculation step of obtaining a degree of imageimprovement of the improved image; and

the second improving step of selecting a degradation parameter withwhich the degree of image improvement becomes highest, and generating animproved image by a degradation function to which the parameter isapplied.

The improving step preferably comprises improving the image whilechanging a value of one parameter at a time, and repeatedly improvingthe image while changing a value of a next parameter after a value ofone parameter with which the degree of image improvement become highestis determined.

Preferably, the method further comprises the segmentation step ofsegmenting the image, the first improving step comprises improving theimage with respect to an image area selected from segmentated areas, andthe second improving step comprises performing improving processing forthe entire image.

Preferably, the method further comprises the selection step of selectingan area, of the areas segmented in the segmentation step, in which acenter of gravity of a frequency distribution corresponds to a highestfrequency, and the first improving step comprises improving the areaselected in the selection step.

Preferably, the method further comprises the input step of inputting avalue of a degradation parameter, and the first improving step comprisesimproving the image by using the degradation function while changing avalue of a degradation parameter included in the degradation functionwithin a predetermined range including a value input in the input step.

Other features and advantages of the present invention will be apparentfrom the following description taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing the schematic arrangement of an imageprocessing system according to the first embodiment of the presentinvention;

FIG. 2 is a flow chart showing the flow of operation of an imagereceiving unit in FIG. 1;

FIG. 3 is a flow chart showing the flow of operation of an informationprocessing apparatus in FIG. 1;

FIG. 4 is a block diagram showing the schematic arrangement of an imageprocessing system according to the second embodiment of the presentinvention;

FIG. 5 is a flow chart showing the flow of operation of an imagereceiving unit in FIG. 4;

FIG. 6 is a flow chart showing the flow of operation of an informationprocessing apparatus in FIG. 4;

FIG. 7 is a block diagram showing the schematic arrangement of an imageprocessing system according to the third embodiment of the presentinvention;

FIG. 8 is a flow chart showing the flow of operation of an imagereceiving unit in FIG. 7;

FIG. 9 is a flow chart showing the flow of operation of an informationprocessing apparatus in FIG. 7;

FIG. 10 is a block diagram showing the schematic arrangement of an imageimproving system according to the fifth embodiment;

FIG. 11 is a flow chart showing an image improving method using theimage improving system of the fifth embodiment;

FIG. 12 is a flow chart showing an image improving method using an imageimproving system according to the sixth embodiment;

FIG. 13 is a flow chart showing an image improving method using an imageimproving system according to the seventh embodiment;

FIG. 14 is a block diagram showing an image processing system accordingto the eighth to 10th embodiments;

FIG. 15 is a flow chart showing a procedure for image improvement in theeighth embodiment;

FIG. 16 is a view showing how a repetitive computation image is cut;

FIG. 17 is a view showing how a repetitive computation image is cut;

FIG. 18 is a flow chart showing a procedure for area selection in theninth embodiment; and

FIG. 19 is a flow chart showing a procedure for image improvement in the10th embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The preferred embodiments of the present invention will be describedbelow.

[First Embodiment]

FIG. 1 shows the schematic arrangement of an image processing systemaccording to the first embodiment of the present invention. For example,an image sensing apparatus 100 is a digital camera and includes an imagesensing unit 110 and a recording medium 120.

For example, the image sensing unit 110 is comprised of an opticalsystem such as lenses, an image sensing element such as a CCD sensor,and the like, and records image information related with a sensed imageon the recording medium 120. For example, the recording medium 120 is ahard disk, flash memory, or the like.

For example, an image receiving unit 210 is controlled by a driver suchas a TWAIN driver. The image receiving unit 210 reads out imageinformation from the recording medium 120, performs predetermined imageprocessing for an image related with the image information, and suppliesthe resultant information to an information processing apparatus 200.

In general, the image processing in the image receiving unit 210includes color interpolation, optical correction, and the like, providedthat the image sensing apparatus 100 is a single-plate digital camera.In this case, for the sake of simplicity, only gray scale conversionsuch as gamma correction and color conversion will be described as theimage processing in the image receiving unit 210.

The information processing apparatus 200 improves an image (degradedimage) related with the image information supplied from the imagereceiving unit 210 to generate an image with little degradation inaccordance with an instruction given by a user through an input unit 220constituted by a keyboard, mouse, and the like. For example, theimproved image is stored in a recording unit 230 or output to an outputunit 240. As the output unit 240, for example, a display or printer canbe suitably used.

FIG. 2 is a flow chart showing the flow of operation of the imagereceiving unit 210. For example, the processing shown in this flow chartis executed by a CPU (not shown) on the basis of a program stored in amemory (not shown). First of all, in step S201, the image receiving unit210 reads out image information from the recording medium 120. Thisimage information is image information of an image that has beendegraded in image sensing operation. In step S202, the image receivingunit 210 performs conversion processing for the image information. Thisconversion processing includes, for example, gamma correction and colorconversion. In step S203, the image receiving unit 210 transfers theconverted image information to the information processing apparatus 200.

FIG. 3 is a flow chart showing the flow of operation of the informationprocessing apparatus 200. Note that the processing shown in this flowchart is executed by the CPU (not shown) on the basis of a programstored in the memory (not shown).

First of all, in step S301, the information processing apparatus 200acquires characteristic information indicating the characteristics ofthe image sensing apparatus 100. For example, this characteristicinformation can be acquired by selecting the characteristic informationof the corresponding image sensing apparatus from the characteristicinformation of a plurality of types of image sensing apparatuses whichare stored in the memory in the information processing apparatus 200 inadvance on the basis of, for example, the apparatus type informationsupplied through the input unit 220.

In step S302, the information processing apparatus 200 acquiresconversion information indicating the contents of conversion processingin the image receiving unit 210. For example, this conversioninformation is stored in the memory in the information processingapparatus 200 when a driver for connecting the image receiving unit 210and the information processing apparatus 200 is installed in theinformation processing apparatus 200.

In step S303, the information processing apparatus 200 determines aconversion method of converting the image information supplied from theimage receiving unit 210. More specifically, the information processingapparatus 200 determines a conversion method on the basis of theconversion information acquired in step S302 (the characteristicinformation acquired in step S301 in addition to the conversioninformation, as needed). The conversion method determined in this caseis a method of converting image information to set an exposure amountand pixel value in a proportional relationship so as to ensure linearityas a precondition for an algorithm for the above image improvingprocessing.

A table in which the types of image sensing apparatuses, the types ofimage receiving units, and conversion methods are made to correspond toeach other may be prepared in the information processing apparatus 200to select an appropriate conversion method by specifying the types ofimage sensing apparatus and image receiving unit. Note that thisconversion method may be determined on the basis of an instruction fromthe user.

The contents (conversion method) of conversion processing on theinformation processing apparatus 200 side in executing gamma correctionon the image receiving unit 210 side will be described in detail below.Letting g(x, y) be the image (degraded image) output from the imagesensing unit 110 of the image sensing apparatus 100, and gg(x, y) be theimage obtained after gamma correction by the image receiving unit 210,the image gg(x, y) is expressed as:

gg(x, y)=g(x, y)^(−1/γ)  (14)

That is, an image with linearity can be obtained by executing aninversion to the processing represented by equation (14). With thisinversion, the image g(x, y) can be reconstructed from the image gg(x,y).

When color correction is to be executed by the image receiving unit 210,the information processing apparatus 200 can obtain an image withlinearity by executing an inversion to the conversion by the colorcorrection. Obviously, when the image receiving unit 210 is to executeboth γ correction and color correction, the information processingapparatus 200 may execute an inversion to the conversion by the γcorrection and an inversion to the conversion by the color correction.There is no need to consider processing other than gray scaleconversions such as color interpolation.

In step S303, for example, a conversion method equivalent to aninversion to the conversion by the image receiving unit 210 isdetermined in the above manner.

In step S304, the information processing apparatus 200 receives imageinformation from the image receiving unit 210. In step S305, theinformation processing apparatus 200 converts the received imageinformation in accordance with the conversion method determined in stepS303.

In step S306, a degradation function H(u, v) is formed on the basis ofthe characteristic information on the image sensing apparatus 100 whichis acquired in step S301. Note that the degradation function isdescribed in “BACKGROUND OF THE INVENTION” with reference to equations(7), (8), (10) and the like.

In step S307, an image f(x, y) with little degradation is reconstructedfrom the image information (image g(x, y)) converted in step S305 on thebasis of the degradation function H(u, v) formed in step S306. Morespecifically, a Fourier transform G(u, v) of the image g(x, y) ismultiplied by 1/H(u, v) to obtain a Fourier transform F(u, v) of theimage f(x, y). An inverse Fourier transform of F(u, v) is then performedto improve the image f(x, y) with little degradation.

First of all, in the above manner, inversion to nonlinear conversiontypified by gray scale conversion such as gamma correction and colorconversion is performed to an input image that has undergone thenonlinear conversion. An inverse transform to the degradation functionis performed for the inverted image information, thereby preventinglarge noise from being superimposed on the improved image.

[Second Embodiment]

FIG. 4 shows the schematic arrangement of an image processing systemaccording to the second embodiment of the present invention. Note thatthe same reference numerals in the second embodiment denote the sameparts as those of the image processing apparatus according to the firstembodiment, and a description thereof will be omitted.

An image receiving unit 210 according to this embodiment includes firstand second conversion units 211 and 212 as a plurality of conversionunits. These conversion units can be selectively used upon switching toconvert the image information read out from a recording medium 120. Thisswitching may be performed in accordance with an instruction given by auser through an operation unit mounted on the image receiving unit 210or an instruction from an information processing apparatus 200.

Assume that the first conversion unit 211 performs conversion processingincluding nonlinear processing (e.g., γ correction and colorcorrection), and the first conversion unit 211 performs conversionprocessing as linear processing.

FIG. 5 is a flow chart showing the flow of operation of the imagereceiving unit 210 in FIG. 4. For example, the processing shown in thisflow chart is executed by a CPU (not shown) on the basis of a programstored in a memory (not shown). First of all, in step S501, the imagereceiving unit 210 reads out image information from the recording medium120.

In step S502, for example, the image receiving unit 210 selects aconversion mode on the basis of an instruction given by the user throughthe operation unit (not shown) mounted on this unit or an instructionfrom the information processing apparatus 200. In step S503, a branch iscaused in accordance with the selected conversion mode. If theconversion mode is the first mode, the flow advances to step S504. Ifthe conversion mode is the second mode, the flow advances to step S505.

In step S504, the first conversion unit 211 converts the imageinformation read out from the recording medium 120. In step S505, thesecond conversion unit 212 converts the image information.

In step S506, the converted image information is transferred to theinformation processing apparatus 200.

FIG. 6 is a flow chart showing the flow of operation of the informationprocessing apparatus 200 in FIG. 4. Note that the processing shown inthis flow chart is executed by the CPU (not shown) on the basis of aprogram stored in the memory (not shown).

First of all, in step S601, the information processing apparatus 200acquires characteristic information indicating the characteristics ofthe image sensing apparatus 100. For example, this characteristicinformation can be acquired by selecting the characteristic informationon the corresponding image sensing apparatus from pieces ofcharacteristic information on a plurality of types of image sensingapparatuses which are stored in the memory in the information processingapparatus 200 on the basis of apparatus type information suppliedthrough the image receiving unit 210.

In step S602, the information processing apparatus 200 acquiresconversion information indicating the contents of conversion processing(image processing) in the image receiving unit 210. For example, thisconversion information is stored in the memory in the informationprocessing apparatus 200 when a driver for connecting the imagereceiving unit 210 and the information processing apparatus 200 to eachother is installed in the information processing apparatus 200.

In step S603, the information processing apparatus 200 receives imageinformation from the image receiving unit 210.

In step S604, it is checked whether the conversion mode of conversionprocessing performed by the image receiving unit 210 is the first orsecond mode. If the first mode is determined, the flow advances to stepS605. If the second mode is determined, the flow advances to step S607.When this conversion mode is to be determined on the image receivingside 210, information indicating a conversion mode and related with thisdetermination is preferably acquired from the image receiving side 210.Note that this information may be obtained through the input unit 220.

In step S605, the information processing apparatus 200 determines aconversion method used to convert the image information supplied fromthe image receiving side 210. More specifically, the informationprocessing apparatus 200 determines a conversion method on the basis ofthe conversion information acquired in step S602 (the characteristicinformation acquired in step S601, as needed, instead of the conversioninformation). The conversion method determined in this case is a methodof converting image information to set an exposure amount and pixelvalue in a proportional relationship so as to ensure linearity as aprecondition for an algorithm for the above image improving processing.More specifically, this method is the same as that used in the firstembodiment.

In step S606, the received image information is converted in accordancewith the conversion method determined in step S605.

In step S607, a degradation function H(u, v) is formed on the basis ofthe characteristic information of the image sensing apparatus 100 whichis acquired in step S301. Note that the degradation function has beendescribed in “BACKGROUND OF THE INVENTION”.

In step S608, an image f(x, y) with little degradation is reconstructedfrom the image information (image g(x, y)) converted in step S606 if theconversion mode is the first mode, or from the image information (imageg (x, y)) received in step S603 if the conversion mode is the secondmode, on the basis of the degradation function H(u, v) formed in stepS607. More specifically, this processing is the same as that in thefirst embodiment.

As described above, when the image receiving side 210 selects aconversion mode to convert image information and executes conversionprocessing (including nonlinear processing) in the first mode, theinformation processing apparatus 200 executes the same processing asthat in the first embodiment. When the image receiving side 210 executesconversion processing (linear conversion processing) in the second mode,the information processing apparatus 200 regards the image informationitself received from the image receiving side 210 as a target forimproving processing.

With this processing, for only an input image having undergone nonlinearconversion typified by gray scale conversion such as gamma correctionand color conversion, inversions to these conversions are performedfirst, and then an inverse transform to the degradation function isperformed for the image, thereby preventing large noise from beingsuperimposed on the improved image. In addition, for an input imagehaving undergone a linear conversion, an inversion thereto is notperformed. Therefore, the time required for the overall improvingprocessing can be shortened, and the load on the processor can bereduced. In this case, since linearity as a precondition for imageimproving processing, no large noise is superimposed on the improvedimage.

[Third Embodiment]

FIG. 7 shows the schematic arrangement of an image processing systemaccording to the third embodiment of the present invention. The samereference numerals in this embodiment denote the same parts as those ofthe image processing apparatus according to the first embodiment, and adescription there of will be omitted.

An image sensing apparatus 100 according to this embodiment includes animage-sensed information recording unit 130 for acquiring informationrelated with image degradation (degradation-related information) such asdegradation parameters (e.g., a shake direction and shake speed)indicating the physical quantities of events that cause imagedegradation, image sensing conditions (e.g., an exposure time, exposureamount, distance to an object, and the focal length of the lens), andthe characteristic information on the image sensing apparatus (e.g., theoptical characteristic of the lens and the identification information ofthe image sensing apparatus), and writing the acquired information on anobjective lens 120. In this case, for example, degradation parametersare detected by an acceleration sensor and the like.

An image receiving side 210 reads out this image-sensed information fromthe objective lens 120, and transfers the information as, for example,additional information of the image-sensed information to an informationprocessing apparatus 200. In addition, the image receiving side 210includes a memory 213 for holding conversion information for specifyingprocessing to be used to convert image information, and transfers thisconversion information as, for example, additional information of imageinformation to the information processing apparatus 200.

In this embodiment, therefore, the information processing apparatus 200can acquire degradation-related information and conversion informationfrom the image receiving side 210.

FIG. 8 is a flow chart showing the operation of the image receiving unit210. Note that the processing shown in this flow chart is executed by aCPU (not shown) on the basis of a program stored in, for example, amemory (not shown).

In step S801, the image receiving unit 210 reads out image informationfrom the recording medium 120. In step S802, the image receiving unit210 performs conversion processing for the image information. Thisconversion processing includes processing such as gamma correction andcolor conversion (nonlinear conversion processing). In this case, as inthe second embodiment, conversion processing may be selectively executedby the image receiving unit 210.

In step S803, the image receiving unit 210 reads out degradation-relatedinformation from the recording medium 120. In step S804, the imagereceiving unit 210 reads out the conversion information held in thememory 213. In this case, if conversion processing is selectivelyexecuted by the image receiving unit 210, information corresponding tothe selected conversion processing is read out from the memory 213.

In step S805, the degradation-related information and conversioninformation are added as pieces of additional information to the imageinformation. More specifically, the image receiving unit 210 convertsthe image information read out from the recording medium 120 into datain a general-purpose format such as the TIFF format or JPEG format, andtransfers the data to the information processing apparatus 200. Manysuch general-purpose formats have header portions on which additionalinformation other than image information can be recorded. Image-sensedinformation and conversion information can therefore be recorded on theheader portion.

In step S806, the image information to which the degradation-relatedinformation and conversion information are added is transferred to theinformation processing apparatus 200. Note that the degradation-relatedinformation and conversion information may not be transferred asadditional information to the image information but may be transferredseparately to the information processing apparatus 200.

FIG. 9 is a flow chart showing the flow of operation of the informationprocessing apparatus 200 shown in FIG. 7. Note that the processing shownin this flow chart is executed by the CPU (not shown) on the basis of aprogram stored in the memory (not shown).

In step S901, the information processing apparatus 200 receives imageinformation to which additional information from the image receivingunit 210 is added. In step S902, the information processing apparatus200 extracts degradation-related information from the image information.In step S903, the information processing apparatus 200 extractsconversion information from the image information.

In step S903, the information processing apparatus 200 determines aconversion method used to convert the image information supplied fromthe image receiving unit 210. More specifically, the informationprocessing apparatus 200 determines a conversion method on the basis ofthe conversion information (the image-sensed information acquired instep S902, as needed, instead of the conversion information) acquired instep S903. The conversion method determined in this case is a method ofconverting image information to set an exposure amount and pixel valuein a proportional relationship so as to ensure linearity as aprecondition for an algorithm for the above image improving processing.More specifically, this method is the same as that used in the firstembodiment.

In step S905, the received image information is converted in accordancewith the conversion method determined in step S904.

In step S906, a degradation function H(u, v) is formed on the basis ofthe characteristic information on the image sensing apparatus 100 whichis acquired in step S902. Note that the degradation function has beendescribed in “BACKGROUND OF THE INVENTION”.

In step S907, an image f(x, y) with little degradation is reconstructedfrom the image information (image g(x, y)) converted in step S905. Morespecifically, a Fourier transform G(u, v) of the image g(x, y) ismultiplied by 1/H(u, v) to obtain a Fourier transform F(u, v) of theimage f(x, Y). An inverse Fourier transform of F(u, v) is then performedto improve the image f(x, y) with little degradation.

According to this embodiment, the information processing apparatus 200acquires the image-sensed information generated by the image sensingapparatus 100 through the image receiving unit 210 and forms adegradation function on the basis of this image-sensed information.Therefore, an improved image nearer to the ideal image can be obtained.In addition, the information processing apparatus 200 acquiredconversion information from the image receiving unit 210, and hence canflexibly cope with an update, change, and the like in the imagereceiving unit 210.

[Fourth Embodiment]

In this embodiment, the processing performed by the informationprocessing apparatus 200 in the first to third embodiments is modified.More specifically, image processing is additionally performed after stepS307, S608, or S907 serving as the improving processing step. As thisadditional image processing (to be referred to as post-processinghereinafter), γ correction, color correction, or the like is suitablyadded. The reason for this will be described below.

In the first to third embodiments, the information processing apparatus200 converts received image information into linear image information.This conversion increases the improving effect, as described above. Onthe other hand, the image corrected by the image receiving unit 210 uponconversion processing is restored to a state similar to the state of theimage sensed by the image sensing apparatus 100. That is, the imageinformation converted by the information processing apparatus 200 isequivalent to the image without γ correction and color correction. Forthis reason, the image represented by this image information looks verydark as a whole. In addition, since the image represented by this imageinformation is the image obtained without correcting the characteristicsof the image sensing apparatus 100, the color gradation, luminancedistribution, and the like are not optimized.

This problem is preferably solved by executing post-processing after theimproving processing. This post-processing is preferably the same as theprocessing performed by an image receiving unit 210. In other words,this processing is preferably equivalent to inversion to the conversionprocessing (step s305, S606, or S905) performed by an informationprocessing apparatus 200. In addition, this post-processing ispreferably optimized on the basis of image-sensed information acquiredas degradation-related information (image sensing conditions, inparticular). Furthermore, the post-processing is preferably optimized inconsideration of characteristic information from an output unit 240.

It is further preferable that the contents of this post-processing beadaptively changed in accordance with the degree and type (e.g., a shakeor blur) of degradation estimated on the basis of degradation-relatedinformation (degradation parameters, in particular). If, for example, itis determined on the basis of degradation-related information that thedegree of degradation is high, the possibility of superimposition oflarge noise is high. In this case, therefore, noise removal processingusing a bandpass filter is preferably added as post-processing. If it isdetermined that the degree of degradation is low, the possibility ofsuperimposition of large noise is low. In this case, therefore, edgeemphasis processing is preferably added as post-processing to attain ahigher image quality.

[Fifth Embodiment]

FIG. 10 is a block diagram showing the schematic arrangement of an imageimproving system according to the fifth embodiment.

An image sensed by an image sensing apparatus 1 such as a camera isformed into an electronic image by an image input apparatus 3 such as afilm scanner by using a silver halide film 2 as a medium. The output ofthe image input apparatus 3 is connected to a computer 4. An inputdevice 5 such as a keyboard, image storage apparatus 6 such as amagnetic disk, and image output apparatus 7 such as a display or printerare connected to the computer 4.

The image sensed by the image sensing apparatus 1 is recorded on thesilver halide film 2. In addition, arbitrary information other than theimage information can be magnetically recorded on the film. Since manytechniques of magnetically recording information on a film have alreadybeen disclosed, a description thereof will be omitted in thisembodiment. The image information and magnetic information recorded onthe silver halide film 2 are loaded into the computer 4 through theimage input apparatus 3. The loaded image is subjected to the processingshown in the flow chart of FIG. 11 in accordance with an instructionfrom the input device 5. The resultant image is displayed on the imageoutput apparatus 7. In addition, the processed image is stored in theimage storage apparatus 6, as needed.

An image improving method using the image improving system will bedescribed next with reference to the flow chart of FIG. 11.

First of all, in step S1101, electronic image information is loaded bythe image input apparatus 3. In step S1102, pixel value conversion isperformed in consideration of the characteristic information on theimage sensing apparatus 1, silver halide film 2, and image inputapparatus 3.

The purpose of the image conversion in step S1102 is to maintainlinearity as a precondition for an image improving algorithm(deconvolution) and perform a conversion so as to set the exposureamount and pixel value in a proportional relationship. Eachcharacteristic information can be used by storing a table of the typesof image sensing apparatus 1, silver halide film 2, and image inputapparatus 3 and corresponding characteristic information in the computer4. In addition, the characteristic information on the image sensingapparatus 1 and silver halide film 2 can be used through a means formagnetically recording information on the silver halide film 2.Characteristic information includes lens aberrations, ISO or the type offilm, film γ value, and the like.

The processing in step S1102 will be described in detail below. For thesake of simplicity, only the gamma characteristic (film γ) of a filmwill be considered as characteristic information. Letting f(x, y) be theimage information proportional to the amount of exposure light incidenton the image sensing plane in the image sensing apparatus, g(x, y) bethe image information after a conversion based on the film γ isexpressed as

g(x, y)=f(x, y)^(−1/γ)  (15)

That is, the image information recorded on the film having this gammacharacteristic is expressed by equation (15). To obtain imageinformation while maintaining linearity, inversion to equation (15) isperformed. When the inversion to equation (15) is performed for theimage information g(x, y) read from the film, the image information f(x,y) proportional to the amount of exposure light can be obtained. In thiscase, characteristics other than the film γ, e.g., lens aberrations andscanner γ, can be properly coped with by sequentially performinginversions upon modeling of characteristic information and conversionsystems.

In step S1103, a degradation function required for image improvingprocessing is formed. Image improving processing is performed in stepS1104 by using the image information obtained in step S1102 and thedegradation function obtained in step S1103.

As described above, according to the image improving system of the fifthembodiment, image conversion is performed to maintain the linearity ofimage information and set the exposure light amount and pixel value in aproportional relationship on the basis of the characteristic informationon the image sensing apparatus, film, and image input apparatus.Thereafter, image improving processing is performed. By using the imageimproving algorithm (deconvolution), therefore, a high-precision imageimprovement can be attained while noise is suppressed to a minimumlevel.

[Sixth Embodiment]

The sixth embodiment will be described next. The schematic arrangementof an image improving system of the sixth embodiment is the same as thatshown in FIG. 10. The same reference numerals in this embodiment denotethe same parts of the fifth embodiment, and a description thereof willbe omitted. In addition to the procedure in the fifth embodiment, thisembodiment includes the step of sensing an image on a silver halide film2 using an image sensing apparatus 1, and magnetically recording imagesensing conditions, sensor outputs, and the like on the silver halidefilm 2 in addition to the characteristic information on the imagesensing apparatus 1 and silver halide film 2. The data recorded on thefilm includes a shake direction, shake speed, and the like detected byan acceleration sensor, image sensing conditions to be set (e.g., anexposure time, exposure light amount, distance to an object, and thefocal length of the lens), characteristic information on the imagesensing apparatus (e.g., the optical characteristics of the lens and theidentification information of the mage sensing apparatus), and the like.These data are used as degradation parameters indicating the physicalquantities of events that cause image degradation. Only processing to beperformed for the image information loaded into a computer 4 will bedescribed below.

FIG. 12 is a flow chart showing the processing performed in the computerin this embodiment.

First of all, in step S1201, electronic image information is loadedthrough the image input apparatus 3. At the same time, the magneticinformation recorded on the silver halide film 2 is read. The magneticinformation includes image sensing conditions and sensor outputs onwhich degradation information is based as well as the characteristicinformation on the image sensing apparatus 1 and silver halide film 2.Consider a blurred image. A more accurate degradation function can beformed by recording a shake path and shake speed during exposure, basedon an exposure time and sensor outputs, as degradation information on ashake. Since detailed degradation information on other types of degradedimages such as an out-of-focus image can be obtained, the effect ofimproving processing to be performed afterward increases.

In step S1202, a pixel value conversion is performed in consideration ofthe characteristic information on the image sensing apparatus 1, silverhalide film 2, and image input apparatus 3. With this operation, theconversion is performed to set the exposure light amount and pixel valuein a proportional relationship.

In step S1203, a degradation function required for image improvingprocessing is formed by using the magnetic information loaded in stepS1201. Image improving processing is performed in step S1204 by usingthe image information obtained in step S1202 and the degradationfunction obtained in step S1203.

According to the sixth embodiment, in addition to the effects obtainedby the fifth embodiment, an image with a higher precision can beobtained by adding degradation information based on image sensingconditions and sensor outputs, together with the characteristicinformation.

[Seventh Embodiment]

The seventh embodiment will be described next. The schematic arrangementof an image improving system according to the seventh embodiment is thesame as that shown in FIG. 10. The same reference numerals in thisembodiment denote the same parts as those in the fifth embodiment, and adescription thereof will be omitted. In addition to the procedure in thefifth embodiment, this embodiment includes the step of sensing an imageon a silver halide film 2 using an image sensing apparatus 1, andmagnetically recording image sensing conditions, sensor outputs, and thelike on the silver halide film 2 in addition to the characteristicinformation on the image sensing apparatus 1 and silver halide film 2.Only processing to be performed for the image information loaded into acomputer 4 will be described below.

FIG. 13 is a flow chart showing the processing performed in the computerin this embodiment.

First of all, in step S1301, electronic image information is loadedthrough an image input apparatus 3. At the same time, the magneticinformation recorded on the silver halide film 2 is read. The magneticinformation includes image sensing conditions, sensor outputs, and thelike, on which degradation information is based, in addition to thecharacteristic information on the image sensing apparatus 1 and silverhalide film 2, as described above.

In step S1302, pixel value conversion is performed in consideration ofthe characteristic information on the image sensing apparatus 1, silverhalide film 2, and image input apparatus 3. With this operation, theconversion is performed to set the exposure light amount and pixel valuein a proportional relationship.

In step S1303, a degradation function required for image improvingprocessing is formed by using the magnetic information loaded in stepS1301. Image improving processing is performed in step S1304 by usingthe image information obtained in step S1302 and the degradationfunction obtained in step S1303.

The improving effect in step S1304 increases owing to the conversion instep S1302. However, several new problems arise. First, since gammacorrection processing and the like are omitted, the resultant imagelooks very dark as a whole. Second, since data is output from aprocessing system totally different from the original processing system,the luminance distribution and color tone of the image differ. Finally,the improving effect varies depending on the degree of degradation, andhence the degree of noise superimposed on image information varies.

To solve the above problems, image processing is performed in step S1305in consideration of magnetic information including the image informationobtained in step S1304. For example, the above problems that the imagebecomes dark and the luminance distribution varies can be solved byperforming inversion to the pixel value conversion performed in stepS1302 for the improved image obtained in step S1304. In addition, byusing the characteristic information on each image output apparatus,more effective image formation can be performed by image processingsuited to the characteristics of each apparatus. More specifically, animage with higher quality can be output by performing a gamma conversioncorresponding to the gamma characteristics of the printer or display towhich image information is output.

In addition, image processing to be performed can be adaptively changedby estimating the degree or type (a shake or blur) of degradation on thebasis of the degradation information included in the magneticinformation loaded in step S1301. More specifically, if, for example, itis determined on the basis of the degradation information that thedegree of degradation is high, the noise superimposed on the imageinformation is large. In this case, therefore, noise removal processingusing a bandpass filter can be additionally performed aspost-processing. If it is determined that the degree of degradation islow, the noise superimposed on the image information is small. In thiscase, therefore, edge emphasis processing or the like can be performedto increase the image improving effect. In this manner, post-processingcan be easily and adaptively added on the basis of degradationinformation.

As described above, according to the seventh embodiment, in addition tothe effects obtained by the sixth embodiment, an image with a higherprecision can be obtained by performing the image processing in stepS1305.

[Eighth Embodiment]

FIG. 14 is a block diagram showing the arrangement of the eight to 10thembodiments. Referring to FIG. 14, an image input apparatus 141 such asa film scanner or digital camera is connected to a computer 142. Theimage input apparatus 141 is used to input image data to the computer142. An input device 143 such as a keyboard, image display apparatus 144such as a display, and image storage apparatus 145 such as a magneticdisk are connected to the computer 142.

An image loaded into the computer 142 through the image input apparatus141 is subjected to the processing shown in the flow chart of FIG. 15 inaccordance with instructions from the input device 143. The resultantimage is displayed on the image display apparatus 144. The processedimage is also stored in the image storage apparatus 145, as needed.

The flow chart of FIG. 15 will be described next.

First of all, in step S1501, an image (to be referred to as a repetitivecomputation image) for an image correction computation to berepetitively performed in the subsequent steps, i.e., steps S1502 toS1504, is generated. As the repetitive computation image in this step,an input degraded image itself may be simply used. Alternatively, animage that can be effectively used to obtain optimal degradationparameters (to be described below) or an image that can suppress thecomputation load may be generated.

If the degradation function obtained by k degradation parametersexpressed by p1, p2, . . . , pk is expressed by H(p1, p2, . . . , pk),image improving processing is performed for the repetitive computationimage generated in step S1501 by using the degradation function H(p1,p2, . . . , pk). For example, with regard to a blurred image, animproved image is generated by using an image improving algorithm with aWiener filter or the like on the basis of a degradation functionobtained by substitutions of a shake angle θ, shake speed V, andexposure time in equations (8) and (9) described in “BACKGROUND OF THEINVENTION”. With regard to an out-of-focus image, an improved image isgenerated by using the same algorithm on the basis of a degradationfunction obtained by substitutions of r and σ in equation (10).

In step S1503, the degree of image improvement is computed from theimproved image obtained in the previous step. The most general case inwhich as an index indicating the degree of image improvement, a powerspectrum (intensity distribution) based on a Fourier transform is usedin consideration of the frequency characteristics of an improved imagewill be described. In general, in a degraded image (blurred image,out-of-focus image, or the like), edge portions (high-frequencycomponents), at which changes in luminance should be acute, becomesmoderate (low-frequency component) due to degradation. The low-frequencycomponents due to the degradation are restored to high-frequencycomponents by the image improving algorithm. For this reason, the degreeof image improvement of an improved image can be regarded as higher withan increase in the number of high-frequency components on the frequencydistribution. Letting R(x, y) be the real part of the Fourier transformimage of an improved image g(x, y) and I(x, y) be the imaginary part, apower spectrum P(x, y) is given by

P(x,y)=R ²(x,y)+I ²(x,y)  (16)

In this case, letting (x₀, y₀) be the center coordinates of an imageP(x, y) and P(r) be the luminance of a pixel at a distance r from thecenter coordinates, a center of gravity C on the frequency distributionis expressed as $\begin{matrix}{C = {\frac{\sum\quad {{rP}(r)}}{\sum\quad {P(r)}}\quad \left( {r = \sqrt{\left( {x - x_{0}} \right)^{2} + \left( {y - y_{0}} \right)^{2}}} \right)}} & (17)\end{matrix}$

This indicates that as the calculated center of gravity C increases, thecenter of gravity is located on a high frequency on the frequencydistribution.

Although the center of gravity in the entire frequency region of thepower spectrum P(x, y) is obtained according to equation (17), efficientoperation can be performed by obtaining the center of gravity in aspecific frequency band in some case. For example, it is known that inan image improving algorithm based on deconvolution using a Wienerfilter or the like, noise due to a parameter error or the like appearsin a high-frequency component of an improved image. In this case, as anindex representing the degree of image improvement, the center ofgravity in a specific frequency range except for a high-frequency regionincluding many noise components is preferably obtained. If the range inwhich the center of gravity on a frequency distribution is obtained isdefined as a≦r≦b, equation (17) is rewritten into $\begin{matrix}{C = {\frac{\sum\quad {{rP}(r)}}{\sum\quad {P(r)}}\quad \left( {{a \leq r \leq b},{r = \sqrt{\left( {x - x_{0}} \right)^{2} + \left( {y - y_{0}} \right)^{2}}}} \right)}} & (18)\end{matrix}$

It is therefore determined that the degree of image improvementincreases with an increase in the center of gravity C′ in the specificfrequency range.

The operation in steps S1502 to S1504 is repeated while a degradationparameter is shifted in step S1504. As a method of shifting adegradation parameter, a method of obtaining an approximate value with alarge step width first, and then calculating a value around theapproximate value with a small step width is available. In addition, theoperator can designate a step width for a shift. In step S1505, adegradation parameter for the highest degree of image improvement isobtained from the result obtained by the operation in steps S1502 toS1504. Finally, in step S1506, an improved image is generated from theinput degraded image by using the degradation function based on thedegradation parameter obtained in step S1505 and an image improvingalgorithm based on a Wiener filter or the like.

If there are a plurality of degradation parameters, one parameter isshifted at a time. If the highest degree of image improvement isobtained when a given parameter assumes a given value, another parameteris shifted. An improved image is generated and the degree of imageimprovement is obtained while the parameter is shifted, therebydetermining a parameter value with which the highest degree of imageimprovement can be obtained. This operation may be repeated for eachparameter.

With the above procedure, even if the value of a degradation parameteris unknown, the value of the degradation parameter can be determined tomake an improved image have the highest quality, and a degraded imagecan be optimally improved by using the parameter. Even if, therefore,degradation parameters in image sensing operation, e.g., informationsuch as an acceleration exerted on the camera itself and image sensingconditions such as an exposure time and exposure light amount, are notprepared for image improvement, a high-quality image can be obtained.

In addition, since no degradation parameters need to be prepared, animproved image can be obtained from not only an image degraded due to acamera shake itself or the like but also an image degraded due to themovement of an object.

[Ninth Embodiment]

In this embodiment, a repetitive computation image generation blockshown in step S1501 in the flow chart of FIG. 15 will be described indetail.

In the eighth embodiment, an input degraded image is simply used as arepetitive computation image. In this case, however, as the image sizeincreases, the computation load greatly increases, and the processingfor the purpose of obtaining a value representing the degree of imageimprovement becomes redundant. In this embodiment, therefore, arepetitive computation image is obtained by converting an input degradedimage having a size S_(x)×S_(y) into an image having a size s_(x)×s_(y)by, for example, thinning out pixels. Many existing techniques areavailable as algorithms for converting the image size S_(x)×S_(y) intothe image size s_(x)×s_(y). For example, the nearest neighbor method,bi-linear method, or bi-cubic method can be used. According to thismethod, an image having undergone a size conversion has almost the samefrequency characteristics as those of an input degraded image, and cancomplete repetitive computations with a small computation amount. Arepetitive computation image size may be fixed in advance or designatedby the operator.

Alternatively, the operator may designate a partial area of an inputimage, and an image cut from the designated area may be used as arepetitive computation image. Assume that the image shown in FIG. 16 isdegraded (out-of-focus, blur, or the like). In this case, a change infrequency characteristic due to the degradation is small in an area Abecause a change in luminance is moderate. At an edge portion in an areaB, however, an acute luminance change (high-frequency component) becomesmoderate (low-frequency component) due to the degradation. As describedabove, when a degraded image is improved by cutting a partial areatherefrom, the improving effect seems to increase as the center ofgravity on the frequency distribution of the improved image shifts tothe high-frequency side. That is, when images are compare in terms ofthe degree of image improvement as in the present invention, the area Ais more suited for the determination of the high/low degree of imageimprovement than the area B. As is obvious from the above description,when the operator designates an area including an edge portion, and animage cut from the area is used as a repetitive computation image, adegradation parameter for the highest degree of image improvement can beeffectively obtained with a light computation load.

A technique of reducing the inconvenience which the operator sufferswhen he/she designates an area in the above technique will be describedbelow. FIG. 17 shows an example of an input degraded image having theimage size S_(x)×S_(y). This input degraded image is divided into afinite number of areas (3×4 areas in FIG. 17) as indicated by the dottedlines in FIG. 17. Assume that each area has the image size S_(x)×S_(y).The number of segmentated areas, the image size of each area, and thelike may be arbitrarily designated. The operator selects the mosteffective area, i.e., the area including most high-frequency components,from a plurality of segmentated areas, and sets the selected area as arepetitive computation image. In addition, a frequency distribution anda class separation degree in the discriminant analysis method may beobtained in units of areas to estimate an area including many edgeportions, and the estimated area may be automatically selected as arepetitive computation image.

FIG. 18 shows a procedure for selecting a repetitive computation imagefrom a degraded image. This procedure is executed in step S1501 in FIG.15 to generate a repetitive computation image.

First of all, in step S1801, an input degraded image is divided intoareas each having a predetermined size. In step S1802, a spatialfrequency distribution in each area is calculated, and the center ofgravity of the distribution is obtained. In step S1803, the area inwhich the calculated center of gravity corresponds to the highestfrequency is selected as a repetitive computation image.

By selecting a part of the image as a repetitive computation image inthis manner, the computation amounts in steps S1502 and S1503 in FIG. 15are reduced to complete the processing quickly.

In addition, by indicating the degree of image improvement with thefrequency characteristics of the improved image, the degree of imageimprovement can be obtained as a numerical value, thereby calculating anoptimal degradation parameter without the mediacy of the operator.

Consider an image that is partially blurred, e.g., an image that ispartially blurred owing to the movement of an object before thebackground although the background is not blurred. The blurred portionof even such an image can be improved. For this purpose, this image issegmented into small areas as shown in FIG. 17, and an area includingthe blurred object is selected as a repetitive computation image. Thisarea is selected by the operator or the like. By executing the procedurein step S1502 and the subsequent steps, the blurred object in the imagecan be improved into a good image.

[10th Embodiment]

The eighth embodiment is based on the premise that degradationparameters are unknown when an input degraded image is to be improved.In this case, since computations are performed by using all degradationparameters in steps S1502 to S1504, a very large load is exerted on thesystem. In addition, large errors occur in obtained degradationparameters depending on the image to be improved, and hence the imageimproving effect may be little.

In this embodiment, a degradation parameter is input or estimated beforethe procedure in FIG. 15 in the eighth embodiment. When a likelydegradation parameter is supplied, image improvement operation can beperformed with a higher precision at a higher speed than in the eighthembodiment.

FIG. 19 is a flow chart in this embodiment.

First of all, in step S1901, a degradation parameter is input/estimatedby the following techniques:

(1) inputting a degradation parameter from the operator;

(2) estimating a degradation parameter on the basis of an output from ameasurement device or the like mounted in the image sensing apparatus;and

(3) estimating a degradation parameter by image analysis on a degradedimage.

According to technique (1), if an infinitely small bright point ispresent in an image, the operator may input a degradation parameter onthe basis of the spread of the bright point. According to technique (2),a degradation parameter can be estimated from image sensing conditionssuch as the exposure and shutter speed of the image sensing apparatus oran output from the measurement device (such as an acceleration sensor)attached to the image sensing apparatus. According to technique (3), adegradation parameter is estimated on the basis of the image featureamount (auto-correlation function or the like) of an input degradedimage. Many such techniques have been used.

In steps S1902 to S1904, the same processing as that in steps S1501 toS1503 in the flow chart (FIG. 15) in the eighth embodiment is performed.In the degradation parameter shift block in step S1905, assuming thatthe optimal degradation parameter to be obtained is near the input(estimated) degradation parameter, the degradation parameter is shiftedonly around the input degradation parameter. In this case, since therange of intervening errors varies depending on the techniques ofinputting (estimating) a degradation parameter (techniques (1) to (3)),a shift range is variably set in accordance with the technique by whicha degradation parameter is input. Alternatively, the operator maydesignate a range.

Finally, in steps S1906 and S1907, a degradation parameter for thehighest degree of image improvement is obtained, and the input degradedimage is improved by using the degradation parameter, as in steps S1505and S1506 in the flow chart in the eighth embodiment (FIG. 15).

With the above procedure, even if a degradation parameter is input, ahigh-precision image improvement can be made by repeating a computationwithin a range near the degradation parameter. In this case, since thenumber of times the above computation is repeated is limited, improvingprocessing can be performed quickly.

[Other Embodiments]

Note that the present invention may be applied to either a systemconstituted by a plurality of devices or an apparatus consisting of asingle device.

In addition, apparatuses and methods constituted by some constituentelements of all the constituent elements of the apparatuses and methodsaccording to the above embodiments are intended by the inventor of thepresent application.

The functions of the apparatuses according to the above embodiments canalso be achieved by permanently or temporarily incorporating a storagemedium, on which program codes are recorded, into a system or apparatus,and making the computer (CPU or MPU) of the system or apparatus read outthe program codes from the storage medium and executing them. In thiscase, the program code itself read out from the storage medium or thestorage medium itself constitutes the present invention.

As the storage medium for supplying the program code, for example, afloppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM,CD-R, magnetic tape, nonvolatile memory card, ROM, and the like can besuitably used. However, other devices may be used.

The functions of the above-mentioned embodiments may be realized notonly by executing the readout program code by the computer but also bysome or all of actual processing operations executed by an OS (operatingsystem) running on the computer on the basis of an instruction of theprogram code.

Furthermore, the functions of the above-mentioned embodiments may berealized by some or all of actual processing operations executed by aCPU or the like arranged in a function extension board or a functionextension unit, which is inserted in or connected to the computer, afterthe program code read out from the storage medium is written in a memoryof the extension board or unit.

As many apparently widely different embodiments of the present inventioncan be made without departing from the spirit and scope thereof, it isto be understood that the invention is not limited to the specificembodiments thereof except as defined in the appended claims.

What is claimed is:
 1. An image processing method of improving an imagesensed by an image sensing apparatus and processed according to a firstconversion process, comprising: a determining step of determiningwhether or not the first conversion process includes a nonlinearconversion; a conversion step of processing the image according to asecond conversion process inverse to the first conversion process if thefirst conversion process includes the nonlinear conversion; and animproving step of processing the image processed in the conversionprocessing step, according to a function for improving the image.
 2. Themethod according to claim 1, wherein the nonlinear conversion includes agamma correction.
 3. The method according to claim 1, wherein thenonlinear conversion includes a color conversion.
 4. The methodaccording to claim 1, further comprising an inputting step of inputtingthe image with additional information being used to generate thefunction.
 5. The method according to claim 4, wherein the additionalinformation includes information relating to the image sensingapparatus.
 6. The method according to claim 4, wherein the additionalinformation includes information relating to image sensing conditions ofthe image.
 7. The method according to claim 1, further comprising afunction generating step of generating the function if the firstconversion process includes a nonlinear conversion.
 8. An imageprocessing apparatus for improving an image sensed by an image sensingapparatus and processed according to a first conversion process,comprising: a determining unit adapted to determine whether or not thefirst conversion process includes a nonlinear conversion; a conversionunit adapted to process the image according to a second conversionprocess inverse to the first conversion process if the first conversionprocess includes the nonlinear conversion; and an improving unit adaptedto process the processed in the conversion unit, according to a functionfor improving the image.
 9. The apparatus according to claim 8, whereinthe nonlinear conversion includes a gamma correction.
 10. The apparatusaccording to claim 8, wherein the nonlinear conversion includes a colorconversion.
 11. The apparatus according to claim 8, further comprisingan inputting unit adapted to input the image with additional informationbeing used to generate the function.
 12. The apparatus according toclaim 11, wherein the additional information includes informationrelating to the image sensing apparatus.
 13. The apparatus according toclaim 11, wherein the additional information includes informationrelating to image sensing conditions of the image.
 14. The apparatusaccording to claim 8, further comprising a function generating unitadapted to generate the function if the first conversion processincludes a nonlinear conversion.
 15. A computer-readable storage mediumstoring a program for causing a computer to execute an image processingmethod for improving an image sensed by an image sensing apparatus andprocessed according to a first conversion process, said programcomprising; a determining step of determining whether or not the firstconversion process includes a nonlinear conversion; a conversionprocession step of processing the image according to a second conversionprocess inverse to the first conversion process if the first conversionprocess includes the nonlinear conversion; and an improving step ofprocessing the image processed in the conversion processing step,according to a function for improving the image.
 16. The mediumaccording to claim 15, wherein the nonlinear conversion includes a gammacorrection.
 17. The medium according to claim 15, wherein the nonlinearconversion includes a color conversion.
 18. The medium according toclaim 15, further comprising an inputting step of inputting the imagewith additional information being used to generate the function.
 19. Themedium according to claim 18, wherein the additional informationincludes information relating to the image sensing apparatus.
 20. Themedium according to claim 18, wherein the additional informationincludes information relating to image sensing conditions of the image.21. The medium according to claim 15, further comprising a functiongenerating step of generating the function if the first conversionprocess includes a nonlinear conversion.